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ABSTRACT 

Developing critical thinking is an important factor in education, hi this study, the author defines critical thinking as the 
set of skills and dispositions which enable one to solve problems logically and to attempt to reflect autonomously by 
means of metacognitive regulation of one's own problem-solving processes. To identify the validity and reliability of 
students’ self -assessment, the critical thinking rubric and Toulmin model combination was introduced. At the end of the 
lesson all Toulmin model work sheets written by students were revised correctly. It was discovered that students solve 
problems logically. This shows that a high level of critical thinking among students is reliable and valid. 
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1. INTRODUCTION 

The ATCS21 (Assessing and Teaching 21st Century Skills) Project proposes ways of thinking; tools for 
working; ways of working; and ways of living in the world as the skills needed for the 21st century. 
According to ATCS21, critical thinking is one aspect of ways of thinking. Developing critical thinking is an 
important factor in education. A number of researchers such as Dewey (1910), Glaser (1941), and Ennis 
(1985) define critical thinking as reflective and logical thinking. 

In this study, the author defines critical thinking as the set of skills and dispositions which enable one to 
solve problems logically and to attempt to reflect autonomously by means of metacognitive regulation of 
one's own problem-solving processes. AACU also provides a rubric known as a value rubric as a critical 
thinking assessment tool. In order to develop their critical thinking skills, it is important for students to be 
able to use this rubric and assess themselves. 

In Gotoh (2016) it was suggested that metacognitive regulation with this critical thinking rubric as the 
criterion could enhance students' critical thinking ability (Gotoh2016). On the other hand, the following two 
problems remain. The first problem concerns the validity and reliability of students’ self-assessment. 
Basically, the achievement level of critical thinking using the critical thinking rubric is dependent on 
students’ judgment. To identify the validity and reliability of students’ own self-assessment, some kind of 
tool is needed. At this point, the author introduces the Toulmin model as a tool to evaluate students’ level of 
logical organization. The Toulmin model consists of six parts: ground, claim, warrant, backing, rebuttal and 
qualifier. If students’ logical organization is deficient, even high self-assessments are not reliable and valid. 

Incidentally, introducing the Toulmin model will assist students in their problem- solving. Will students 
feel that the combination of the critical thinking rubric and the Toulmin model is useful? In this study, the 
author considers this point. 

The second problem concerns differences among individuals. In particular, nobody yet knows the 
difference in critical thinking achievement levels between those who have an aptitude for critical thinking 
and those who do not. Comparative study is needed.A template is a set of styles and page layout settings that 
determine the appearance of a document. 
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2. METHOD 


2.1 Knowledge-constructive Jigsaw Method 

27 university students took part in the study. Cooperative problem-solving methods such as the 
knowledge-constructive jigsaw method were selected as materials. The research project consisted of the 
knowledge-constructive jigsaw method and metacognitive regulation. Students held a discussion on the 
subject of introducing non-Japanese workers into Japan. Four types of different information were provided 
separately: depopulation in Japanese society; disadvantages and difficulties of employing non-Japanese 
workers; an overseas case study; and trends among non-Japanese workers in selecting an employment 
destination. After reading these different types of information individually, the students discussed the 
following topic: “Should Japan bring in more foreign workers?” 

The entire discussion was audio-recorded and a lesson protocol was developed to be used in subsequent 
metacognitive regulation and reflection. Students were asked to carry out self-assessment using the critical 
thinking rubric (Gotoh2016) . 

• I pay attention to the information source (who wrote it). 

• I pay attention to the information destination (who reads it). 

• I pay attention to the information purpose (agenda). 

• I assume information from an opposing point of view. 

• I pay attention to information period (when it was produced). 

• I pay attention to inconsistencies and missing information. 

• I pay attention to gaps in the argument. 

• I make an impartial valuation of information that goes against my own opinion. 

• If necessary, I reserve judgment. 

• If necessary, I make a conditional judgment. 

Self-assessment was carried out while listening to the audio recording of the discussion and reading the 
lesson protocol. Students were asked to self-assess their critical thinking ability on three levels: achieved, 
partly achieved, failed. Students were asked to explain their logic using the Toulmin model work sheet. The 
Toulmin model consists of six parts: ground, claim, warrant, backing, rebuttal and qualifier. The validity or 
non-validity of students’ logic was evaluated according to whether they could correctly describe the 
relationship among claim, ground and warrant. Figures should de numbered consecutively as they appear in 
the text. 

2.2 Metacognitive Regulation and Reflection 

Students were asked to reflect on their learning using the critical thinking rubric and a rating scale, and by 
writing comments. These activities are themselves metacognitive regulation. Items on the rating scale are as 
follows: 

• Using the critical thinking rubric and Toulmin model, it is easy to make a correct judgment. 

• Using the critical thinking rubric and Toulmin model, it is easy to organize a discussion. 

• Using the critical thinking rubric and Toulmin model, it is possible to clarify the problem. 

• The critical thinking rubric and Toulmin model are useful in daily life. 

The critical thinking rubric and Toulmin model are difficult to use. In order to focus on critical thinking 
dispositions, the Critical Thinking Disposition Scale (Hirayama & Kusumi, 2004) was used. This scale 
consists of four factors: awareness of logical thinking, spirit of inquiry, objectivity and emphasis on 
evidence. The scores of 33 items on this 4-point scale were totaled and the top two students and lowest two 
students were extracted. 
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Comparison and analysis of the top 50% and lowest 50% and the top 25 % and lowest 25% were also 
carried out. 


3. RESULTS 


3.1 Achievement Levels in the Critical Thinking Rubric 

Figure 1 shows achievement levels in the critical thinking rubric. Over 90% of students had a high level of 
achievement in the following items: “I make an impartial evaluation of information that goes against my 
own opinion,” “If necessary, I reserve judgment,” I pay attention to the information source (who wrote it)” 
”1 pay attention to inconsistencies and missing information,” and ”1 assume information from an opposing 
point of view.” Over 70% of students had a high level of achievement in the following items: ”1 pay 
attention to the information period (when it was produced)” and ”1 pay attention to the information 
destination (who reads it).” In comparison with the above, there was a lower level of achievement in the 
following items: “I pay attention to the information source (who wrote it)” and “I pay attention to gaps in the 
argument.” 

I MAKE AN IMPARTIAL VALUATION OF INFORMATION THAT GOES AGAINST MY OWN 

OPINION. 3.7^| 

IF NECESSARY,! R ES ER VE J U DG M E NT. 7.4 

I PAY ATTENTION TO THE INFORMATION P U R POSE (AG E ND A). 7.4 

I ASSUME INFORMATION FROM AN OPPOSING POINT OF VIEW. 11.1 

I PAY ATTENTION TO INCONSISTENCIES AND MISSING INFORMATION. 11.5 

I MAKE AN IMPARTIAL VALUATION OF INFORMATION THAT GOES AGAINST MY OWN 

OPINION. 148 

I PAY ATTENTION TO INFORMATION PERIOD (WHEN IT WAS PRODUCED). 29.6 

I PAY ATTENTION TO THE INFORMATION DESTINATION (WHO READS IT). 29.6 

I PAY ATTENTION TO THE INFORMATION SOURCE (WHO WROTE IT). 37.0 

I PAY ATTENTION TO GAPS IN THE ARGUMENT. 38.5 

Failed. ■ Partly achieved ■ Achieved 

Figure 1. Achievement levels in the critical thinking rubric 

3.2 Evaluation of Logical Organization 

The relationship among ground, claim and warrant was evaluated using the Toulmin model work sheet. At 
first, some students described the relationship among these three components incorrectly. However, at the 
end of the lesson, as a result of discussions with each other, all the Toulmin model work sheets were revised 
correctly. Evaluation of the critical thinking rubric and Toulmin model 

Figure 2 shows the evaluation of the critical thinking rubric and Toulmin model. Using the critical thinking 
rubric and Toulmin model, about 90% of students feel “It is easy to make a correct judgment,” “It is easy to 
organize a discussion,” and ”It is easy to clarify the problem.” On the other hand, students also feel that the 
critical thinking rubric and Toulmin model is hard to use (about 50%) and difficult to use in daily life (about 
40%). 
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IT IS EASY TO MAKEACORRECT JUDGMENT 


IT IS EASY TO ORGANIZE A DISCUSSION. 


IT IS POSSIBLETO CLARIFYTHE PROBLEM 


USEFUL IN DAILY LIFE. 


DIFFICULT TO USE 



Strongly Disagree Disagree ■ Agree ■ Strongly Agree 


Figure 2. Evaluation of the critical thinking rubric and Toulmin model 


3.3 Contents Analysis of Evaluation of Critical Thinking Rubric and Toulmin 
Model 

In comparison with Gotoh(2016), which did not use the Toulmin model, many students referred to logical 
organization and rebuttal. I realized that in daily life, I never consider evidence for information rebuttal and I 
always swallow information uncritically. Considering rebuttal evidence is useful in judging the reliability of 
information. Using the Toulmin model, I can clearly focus on the crux of the problem. In comparison with 
writing about claim and ground, it takes me much longer to write a rebuttal so I realize I am not very good at 
thinking about something from different points of view. 

3.4 Comparison of Critical Thinking Dispositions 

Critical thinking disposition score was calculated (average score 111.18, max score 153, minimum score 84). 
The top 2 students and lowest 2 students, the top 50% and lowest 50% and the top 25 % and lowest 25% 
were extracted. Three types of comparison studies (a comparison of the top 2 students and lowest 2 students, 
the top 50% and lowest 50% and the top 25 % and lowest 25%) were carried out. As a result of cross 
tabulation of achievement levels in the critical thinking rubric, no significant difference was found in any 
comparisons (Cramer V ns.). In the evaluation of the critical thinking rubric and Toulmin model, a 
significant difference was found between the top 2 students and the lowest 2 students. In the case of high 
critical thinking dispositions, the critical thinking rubric and Toulmin model tended to be useful for the item 
about clarifying the crux of the problem (Cramer V.046). No significant difference was found among any 
groups (Cramer V ns.). 


4. CONCLUSION 

To identify the validity and reliability of students’ self -assessment, the critical thinking rubric and Toulmin 
model combination was introduced. At the end of the lesson all Toulmin model work sheets written by 
students were revised correctly. It was discovered that students solve problems logically. This shows that a 
high level of critical thinking among students is reliable and valid. 
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It was also discovered that students are aware of the importance and usefulness of considering rebuttal 
evidence using the critical thinking rubric and Toulmin model combination. 

On the other hand, there was no clear difference between high and low dispositions for critical thinking. 

In this research the sample size was extremely limited. In the near future, the author hopes to collect a 
large data sample. Differences among individuals, such as differences in personality and knowledge will also 
be taken into consideration. 

ACKNOWLEDGEMENT 

This work was supported by JSPS KAKENHI Grant Number 15K01020. 


REFERENCES 

Assessment & teaching of 21st century skills 2015 21st skills, http://www.atc21s.org/ 

Dewey,J. 1933.How we think (Revised and expanded edition). MA: Houghton Mifflin Company. 

Ennis,R.H. 1985 A logical basis for measuring critical thinking skills. Educational Leadership, 43, 44-48 
Gotoh,Y. 2016 Development of Critical Thinking with Metacognitive Regulation. Proceedings of 13th international 
Conference on Cognition and Exploratory Learning in Digital Age(CELDA2016), 353- 356 
Gotoh,Y. 2015 Development of Critical Thinking Self-Assessment System using Wearable Device. Proceedings of 12th 
international Conference on Cognition and Exploratory Learning in Digital Age(CELDA2015), 387- 389 
Gotoh,Y. 2013 Visualization of Understanding of Media Characteristics Using Analytic Hierarchy Process. Computer 
Technology and Application, Vol.4, pp.563-567 

Graser, E.M. 1941 An experiment in the development of critical thinking. New York: Teachers College of Columbia 
University, Bureau of Publications. 

Hirayama,R. & Kusumi,T.2004 Effect of Critical Thinking Disposition on Interpretation of Controversial Issues : 
Evaluating Evidences and Drawing Conclusions. Japan Journal of Educational Psychology, Vol.52(2),186-198 in 
JAPANESE 

Ministry of education, culture, sports science and technology Japan 2015 Result of National Assessment of Information 
Utilization Ability in Japan, http://www.mext.go.jp/a_menu/shotou/zyouhou/1356188.htm 
The Association of American Colleges and Universities 2015 Critical thinking value rubric 
https://www.aacu.org/value/rubrics/critical-thinking 


285 



